On the capacity and superposition of minima in neural network loss function landscapes

Abstract Minima of the loss function landscape (LFL) a neural network are locally optimal sets weights that extract and process information from input data to make outcome predictions. In underparameterised networks, capacity may be insufficient fit all relevant information. We demonstrate different local minima specialise in certain aspects learning problem, differently. This effect can exploited using meta-network which predictive power multiple LFL is combined produce better classifier. With this approach, we increase area under receiver operating characteristic curve by around 20 % for complex problem. propose theoretical basis combining show how trained select representative used classification specific item. Finally, present an analysis symmetry-equivalent solutions machine problems, provides systematic means improve efficiency approach.